Running a Simulation

SPECT3D simulations can be started using the user interface or, alternatively, can be run in batch mode.

There are several parallel processing options that can be utilized for SPECT3D runs:

• Multi-threaded option: suitable for single platform multi-core machines (parameters set in Preferences).
• MPI option: suitable for Linux clusters. The MPI parallelization option is supported only in batch mode.

During a SPECT3D run, a number of output files are generated, which are written to the "working directory". The default location of the working directory is initially set based on the Preferences settings. The user can also choose an alternative directory.

When starting a run using the user interface, the user supplies the working directory and run name. To start the run, click the Run button.

At the beginning of the run, the workspace parameters will be checked for consistency. If input parameters are satisfactory, the Progress Window will appear.

Running Simulations in Batch Mode

Simulations can be run in batch mode using a saved workspace. The command line arguments are:

-b Run in batch mode (required).

-i WORKSPACE Input workspace file (required).

-o RUNNAME Name for the output files (optional, defaults to workspace name).

-d OUTPUT_DIR The directory for the output files (optional, defaults to workspace directory, the directory must exist).

-x Overwrite existing data file (optional).

-e HYDRO FILE The hydro file to use (optional, defaults to the file listed in the workspace)

-n NUMBER OF PROCESSORS Number of processors to use during simulation (optional). If set, this overrides the value read from the preferences file.

-h Prints this help message.

-p MPI parallelization options.*

For example:

-b -i /test_path/test_workspace.spw -o test_run -d /results_path -x

Note: in batch mode, user-specified ("Other") paths for atomic data and opacity files are read from the workspace file (.spw) instead of the Preferences file, while the corresponding default paths are still read from the Preferences file. The flag specifying whether to use the user-specified vs. the default path is also read from the workspace in this case (not the Preferences file).

Options for MPI (Message Passing Interface)  

To utilize MPI parallelization for a SPECT3D simulation, use the "-p" option on the command line, followed by an integer specifying the type of parallel processing required. The options are:

• 0: No parallelization
• 1: Parallelization is performed over photon energies when computing opacities and updating radiative rate integrals.
• 2: Parallel domain decomposition is performed for 2-D or 3-D Cartesian geometries.
• 3: Parallel decomposition is performed over duplicate grids (e.g., "pie slices" in z-pinch simulations) .
• 4: Parallelization is performed over detector pixels.
• 5: Parallelization is performed over time steps.
• 6: Parallelization of X-ray Scattering calculations is performed.

Parallel domain decomposition can only be invoked with a command line argument while running in batch mode. For example, a command line could be:

mpirun -np 4 Spect3d_mp -b -p 4 -i test.spw

If the MPI parallel option is specified, parallel options set in the Preferences (for multi-core platforms) will be ignored.

Domain Decomposition

For domain decomposition, the number of domains is determined by the number of EXODUS files available for the simulation. The number of processors (-np parameter for mpirun) MUST be equal to the number of domains (EXODUS files). The following naming convention is used for the exodus files:

<any_name>.exo.<total_number_of_processors>.<processor_rank>

For example, for 3 domains, EXODUS file names could be:

my_run.exo.3.0

my_run.exo.3.1

my_run.exo.3.2

For 100 domains, EXODUS file names could be:

my_run.exo.100.000

my_run.exo.100.001

……………………

my_run.exo.100.099

Any of the exodus files can be used to setup a calculation (in *.spw file). SPECT3D automatically generates appropriate suffixes based on the number of available processors and processor rank.

When a calculation is performed with domain decomposition, the drill-down option in VISUALIZER will not be available.

Next